Effect of Dirty Data on Analysis Results

Abstract: Information quality assessment is the process of inspecting business information to ensure that it meets the needs of the knowledge workers who depend on it. We suggest in this paper that, prior to implementing a system to assess quality, those responsible for information quality can use a subset of clean data to create a statistical model of a decision that relies on the information. Simulated perturbations of the clean data can then be used to establish a boundary for determining what degree of error produces erratic, unusable results. This approach has the advantage that it can be used to show the effects of poor data quality on the result of the analysis as accuracy declines for any reason. We do not focus on the reasons why the quality declines but rather show the consequences of poor quality data on the results of the analysis. Moreover, we examine a general error structure, one that is common in situations where errors are not additive, and is more general than that previously considered in the literature,